Statistical method of averaging

[1] 112.2274

Fake Data and the average as an estimate

  • The mean value summarises the path price into a single number
  • But how accurate is this algorithm?
  • The textbook answer is given in terms of the standard error

Standard error function

[1] 0.98525
  • Here averaging is the .content-box-red[algorithm], while the standard error provides the .content-box-blue[inference] of the algorithm’s accuracy.

Explanation

  • It is a surprising, and crucial, aspect of statistical theory that the same data that supplies an estimate can also assess its accuracy.
  • Strictly speaking Inference concerns more than accuracy: recall that algorithms say what the statistician does while inference says why she does it.

Algorithms and Inference

Algorithms and regression

.panelset[ .panel[.panel-name[Least squares algorithm for linear regression] - The least squares estimator is a popular algorithm for estimating a linear regression - The algorithm fits the data by least squares, by minimising the sum of squared deviations over all choices of the model parameters. - Consider the following fake relationship between the price and some market factor

Least squares algorithm

Lowess algorithm for localised regression

  • Lowess is a modern computer based algorithm which works by moving its attention along the x-axis, fitting local polynomial curves of differing degrees to nearby (x,y) coordinates.
  • The fitted estimate above has a similar linear regression as the least squares algorithm in the middle of the data but for higher values of the factor has a much steeper curve.

Hypothesis testing

Two sample t-test inference

  • We obtain t= 5.07 which would classically be considered very strong evidence that the apparent difference is genuine; in standard terminology, “with two-sided significance level 0.0000003.”
  • A small significance level (or “p-value”) is a statement of statistical surprise: something very unusual has happened if in fact there is no difference in returns of the value factor in up and down markets
  • We are less surprised by t=5.07 if this comparison of mean returns is just one candidate out of thousands that might have produced “interesting” results.
  • For example if I take a different set sample period and run the test again, or I split the sample into 5 sub-samples and run the test again.

Traditional hypothesis testing and false discovery theory

  • A primary goal of empirical research is to identify the variables involved in a phenomenon
  • The identification of these variables is a prerequisite to the formulation of a theory
  • In classical statistics (e.g., Econometric), the significance of variables is established through p values
  • p values suffer from multiple flaws, which have led to the acknowledgment that most discoveries in finance are false
  • False discovery rate theory is an impressive advance in statistical inference, incorporating Bayesian ,frequentist, and empirical Bayesian elements.
  • It was a necessary advance in a scientific world where computer-based technology routinely presents thousands of comparisons to be evaluated at once.

Simulated example of false discovery problem

set.seed(1235)
res<-tibble(testno=1:20, pvalue=1)
for (i in 1:20) {
  Up=rnorm(1000)
  NoUp=rnorm(1000)
  res[i,2]<-t.test(Up,NoUp)['p.value']
}
  • This code fakes the return up versus down question earlier, where the ground truth is that there is no relationship.
  • This implies that our significant results, when using a t-test, must be fake
  • We repeat the test 20 times using a simple for loop.
  • Arranging the results in ascending order of p-values.
  • With enough hypothesis test we will always discovery a fake results.
  • In financial machine learning, where we have to work with high-dimensional data with many features, 1000’s of comparisons are common.
LS0tCnRpdGxlOiAiQ29tcHV0YXRpb25hbCBub3RlYm9vayBmb3IgUGhEIFNlbWluYXIiCm91dHB1dDogaHRtbF9ub3RlYm9vawotLS0KYGBge3Igc2V0LXVwfQprbml0cjo6b3B0c19jaHVuayRzZXQoZWNobyA9IEZBTFNFKQpwYWNtYW46OnBfbG9hZCgieGFyaW5nYW5FeHRyYSIsImtuaXRyIiwia2FibGVFeHRyYSIsImZvbnRhd2Vzb21lIiwidGlkeXZlcnNlIiwieGFyaW5nYW50aGVtZXIiKQpgYGAKCgoKIyBTdGF0aXN0aWNhbCBtZXRob2Qgb2YgYXZlcmFnaW5nCgotIENvbnNpZGVyIHRoZSBzaW1wbGlzdGljIGJ1dCBtb3N0IHBvcHVsYXIgbWVhc3VyZSBvZiBhdmVyYWdpbmcgCi0gU3VwcG9zZSB3ZSBvYnNlcnZlIGEgc2V0IG9mIHRoZSBwcmljZSBjaGFuZ2UgcGF0aCBwaGVub21lbm9uIG9mIGEgc3RvY2sgb3ZlciB0IHBlcmlvZHMgJHhfMSwgeF8yLC4uLix4X3QkLgoKYGBge3IgcmFuZG9tLXdhbGstc2ltdWxhdG9yLCBmaWcuaGVpZ2h0PTQsIGZpZy53aWR0aD04fQpzZXQuc2VlZCgxMjMpClMxPTEwMDsgdD0xMDA7IFByaWNlPXZlY3RvcihsZW5ndGggPSB0KQpub2lzZT1ydW5pZih0LG1pbiA9IC01LG1heCA9IDUpCmZvciAodHQgaW4gMTp0KSB7CiAgaWYgKHR0PT0xKSB7CiAgICBQcmljZVt0dF09UzEKICB9CiAgZWxzZXsKICBQcmljZVt0dF0gPSBQcmljZVt0dC0xXSArIG5vaXNlW3R0XQogIH0KfQpzdW0oUHJpY2UpL2xlbmd0aChQcmljZSkKYGBgCgotIFRoZSBhdmVyYWdpbmcgZnVuY3Rpb24gaGlnaGxpZ2h0ZWQgYWJvdmUgaXMgYW4gYWxnb3JpdGhtIGFuZCBpcyByZXByZXNlbnRlZCBieSB0aGUgcmVkIGxpbmUgaW4gdGhlIHBsb3Qgb3Bwb3NpdGUKCiMjIEZha2UgRGF0YSBhbmQgdGhlIGF2ZXJhZ2UgYXMgYW4gZXN0aW1hdGUKCmBgYHtyIGVjaG89RkFMU0UsIGZpZy5oZWlnaHQ9NSwgZmlnLndpZHRoPTd9CnByaWNlX21lYW49c3VtKFByaWNlKS9sZW5ndGgoUHJpY2UpClByaWNlICU+JSBwbG90KHR5cGU9ImwiKSAlPiUgYWJsaW5lKGg9cHJpY2VfbWVhbiwgY29sPSJyZWQiKQpgYGAKCi0gVGhlIG1lYW4gdmFsdWUgc3VtbWFyaXNlcyB0aGUgcGF0aCBwcmljZSBpbnRvIGEgc2luZ2xlIG51bWJlcgotIEJ1dCBob3cgYWNjdXJhdGUgaXMgdGhpcyBhbGdvcml0aG0/Ci0gVGhlIHRleHRib29rIGFuc3dlciBpcyBnaXZlbiBpbiB0ZXJtcyBvZiB0aGUgKnN0YW5kYXJkIGVycm9yKgoKIyMgU3RhbmRhcmQgZXJyb3IgZnVuY3Rpb24KCmBgYHtyLCB3YXJuaW5nPUZBTFNFfQpzdGFuZGFyZF9lcnJvcjwtIGZ1bmN0aW9uKHgpewogIGR4PSh4LW1lYW4oeCkpXjIKICBkZW5vbT1sZW5ndGgoeCkqKGxlbmd0aCh4KS0xKQogIHN1bXNxPXN1bShkeCkvZGVub20KICBzcXJ0KHN1bXNxKQp9CnN0YW5kYXJkX2Vycm9yKFByaWNlKQpgYGAKLSBIZXJlIGF2ZXJhZ2luZyBpcyB0aGUgLmNvbnRlbnQtYm94LXJlZFthbGdvcml0aG1dLCB3aGlsZSB0aGUgc3RhbmRhcmQgZXJyb3IgcHJvdmlkZXMgdGhlIC5jb250ZW50LWJveC1ibHVlW2luZmVyZW5jZV0gb2YgdGhlIGFsZ29yaXRobSdzIGFjY3VyYWN5LgoKIyMgRXhwbGFuYXRpb24KCi0gSXQgaXMgYSBzdXJwcmlzaW5nLCBhbmQgY3J1Y2lhbCwgYXNwZWN0IG9mIHN0YXRpc3RpY2FsIHRoZW9yeSB0aGF0IHRoZSBzYW1lIGRhdGEgdGhhdCBzdXBwbGllcyBhbiBlc3RpbWF0ZSBjYW4gYWxzbyBhc3Nlc3MgaXRzIGFjY3VyYWN5LiAKLSBTdHJpY3RseSBzcGVha2luZyAqSW5mZXJlbmNlKiBjb25jZXJucyBtb3JlIHRoYW4gYWNjdXJhY3k6IHJlY2FsbCB0aGF0IGFsZ29yaXRobXMgc2F5IHdoYXQgdGhlIHN0YXRpc3RpY2lhbiBkb2VzIHdoaWxlIGluZmVyZW5jZSBzYXlzIHdoeSBzaGUgZG9lcyBpdC4KCiMgQWxnb3JpdGhtcyBhbmQgSW5mZXJlbmNlCi0gT2YgY291cnNlLCB0aGUgYHN0YW5kYXJkX2Vycm9yKClgIGZ1bmN0aW9uIGRlZmluZWQgcHJldmlvdXNseSBpcyBpdHNlbGYgYW4gYWxnb3JpdGhtLCB3aGljaCBjb3VsZCBiZSAoYW5kIGlzKSBzdWJqZWN0IHRvIGZ1cnRoZXIgaW5mZXJlbnRpYWwgYW5hbHlzaXMgY29uY2VybmluZyBpdHMgYWNjdXJhY3kgCgotIC5jb250ZW50LWJveC1ncmVlbltUaGUgcG9pbnQgaXMgdGhhdCB0aGUgYWxnb3JpdGhtIGNvbWVzIGZpcnN0IGFuZCB0aGUgaW5mZXJlbmNlIGZvbGxvd3MgYXQgYSBzZWNvbmQgbGV2ZWwgb2Ygc3RhdGlzdGljYWwgY29uc2lkZXJhdGlvbi5dCgotIC5jb250ZW50LWJveC15ZWxsb3dbSW4gcHJhY3RpY2UgdGhpcyBtZWFucyB0aGF0IGFsZ29yaXRobWljIGludmVudGlvbiBpcyBhIG1vcmUgZnJlZS13aGVlbGluZyBhbmQgYWR2ZW50dXJvdXMgZW50ZXJwcmlzZV0KCi0gLmNvbnRlbnQtYm94LWdyZXlbSW4gY29udHJhc3QgaW5mZXJlbmNlIHBsYXlpbmcgY2F0Y2gtdXAgYXMgaXQgc3RyaXZlcyB0byBhc3Nlc3MgdGhlIGFjY3VyYWN5LCBnb29kIG9yIGJhZCwgb2Ygc29tZSBob3QgbmV3IGFsZ29yaXRobWljIG1ldGhvZG9sb2d5Ll0KCgojIEFsZ29yaXRobXMgYW5kIHJlZ3Jlc3Npb24KLnBhbmVsc2V0WwoucGFuZWxbLnBhbmVsLW5hbWVbTGVhc3Qgc3F1YXJlcyBhbGdvcml0aG0gZm9yIGxpbmVhciByZWdyZXNzaW9uXQotIFRoZSBsZWFzdCBzcXVhcmVzIGVzdGltYXRvciBpcyBhIHBvcHVsYXIgYWxnb3JpdGhtIGZvciBlc3RpbWF0aW5nIGEgbGluZWFyIHJlZ3Jlc3Npb24KLSBUaGUgYWxnb3JpdGhtIGZpdHMgdGhlIGRhdGEgYnkgKmxlYXN0IHNxdWFyZXMqLCBieSBtaW5pbWlzaW5nIHRoZSBzdW0gb2Ygc3F1YXJlZCBkZXZpYXRpb25zIG92ZXIgYWxsIGNob2ljZXMgb2YgdGhlIG1vZGVsIHBhcmFtZXRlcnMuCi0gQ29uc2lkZXIgdGhlIGZvbGxvd2luZyBmYWtlIHJlbGF0aW9uc2hpcCBiZXR3ZWVuIHRoZSBwcmljZSBhbmQgc29tZSBtYXJrZXQgZmFjdG9yCmBgYHtyIGZha2UgZGF0YX0KZmFjdG9yPC1QcmljZSArIHJ1bmlmKHQsMSw4KV4yICM8PAp0aWJibGUoUHJpY2UsZmFjdG9yKS0+ZGYgCmBgYAoKTGVhc3Qgc3F1YXJlcyBhbGdvcml0aG0KCmBgYHtyIGVjaG89RkFMU0V9CmRmICU+JSBnZ3Bsb3QoYWVzKHk9UHJpY2UseD1mYWN0b3IpKSArIGdlb21fcG9pbnQoKSArCiAgZ2VvbV9zbW9vdGgobWV0aG9kID0gImxtIikgIzw8IGFkZHMgbGVhc3Qgc3F1YXJlcyBsaW5lIHdpdGggc3RhbmRhcmQgZXJyb3JzCmBgYAoKLSBUaGlzIGNvZGUgbWFudWZhY3R1cmVzIGEgcG9zaXRpdmUgcmVsYXRpb25zaGlwIHRvIHRoZSBmYWN0b3IgcGx1cyBzb21lIG5vaXNlIGFuZCB0aGVuIGRyYXdzIHRoZSBsZWFzdCBzcXVhcmVzIHJlZ3Jlc3Npb24gbGluZS4KLSBUaGUgYWNjdXJhY3kgb2YgdGhpcyBlc3RpbWF0ZSBpcyBnaXZlbiBieSAkXHBtJCAyIHN0YW5kYXJkIGVycm9ycwotIFRoZSBhcHByb3ByaWF0ZSBpbmZlcmVuY2Ugb2YgdGhpcyBiYW5kZWQgZ3JleSBhcmVhIGlzIHRoYXQgdGhpcyBoYXMgYSA5NSUgY2hhbmNlIGluIGluY2x1ZGluZyB0aGUgdHJ1ZSBleHBlY3RlZCB2YWx1ZSBvZiBgUHJpY2VgIGluIGFuIGBVcGAgbWFya2V0LgotIFRoaXMgOTUlIGNvdmVyYWdlIGRlcGVuZHMgb24gdGhlIHZhbGlkaXR5IG9mIHRoZSBsaW5lYXIgcmVncmVzc2lvbiBtb2RlbCwgd2hpY2ggY291bGQgYXMgZWFzeSBoYXZlIGJlZW4gYSBxdWFkcmF0aWMgcmVsYXRpb25zaGlwCgojIyBMb3dlc3MgYWxnb3JpdGhtIGZvciBsb2NhbGlzZWQgcmVncmVzc2lvbgoKKiBMb3dlc3MgaXMgYSBtb2Rlcm4gY29tcHV0ZXIgYmFzZWQgYWxnb3JpdGhtIHdoaWNoIHdvcmtzIGJ5IG1vdmluZyBpdHMgYXR0ZW50aW9uIGFsb25nIHRoZSB4LWF4aXMsIGZpdHRpbmcgbG9jYWwgcG9seW5vbWlhbCBjdXJ2ZXMgb2YgZGlmZmVyaW5nIGRlZ3JlZXMgdG8gbmVhcmJ5IGAoeCx5KWAgY29vcmRpbmF0ZXMuIAotIFRoZSBmaXR0ZWQgZXN0aW1hdGUgYWJvdmUgaGFzIGEgc2ltaWxhciBsaW5lYXIgcmVncmVzc2lvbiBhcyB0aGUgbGVhc3Qgc3F1YXJlcyBhbGdvcml0aG0gaW4gdGhlIG1pZGRsZSBvZiB0aGUgZGF0YSBidXQgZm9yIGhpZ2hlciB2YWx1ZXMgb2YgdGhlIGZhY3RvciBoYXMgYSBtdWNoIHN0ZWVwZXIgY3VydmUuCgpgYGB7ciBsb2Vzc30KZGYgJT4lIGdncGxvdChhZXMoeT1QcmljZSx4PWZhY3RvcikpICsgZ2VvbV9wb2ludCgpICsKICBnZW9tX3Ntb290aChtZXRob2QgPSAibG9lc3MiKSAjIApgYGAKCgpgYGB7ciBhbmltYXRpb24gY29kZX0Kc2V0LnNlZWQoMTIzKQpTMT0xMDA7IHQ9MTAwOyBQcmljZT12ZWN0b3IobGVuZ3RoID0gdCkKbm9pc2U9cnVuaWYodCxtaW4gPSAtNSxtYXggPSA1KQpmb3IgKHR0IGluIDE6dCkgewogIGlmICh0dD09MSkgewogICAgUHJpY2VbdHRdPVMxCiAgfQogIGVsc2V7CiAgICBQcmljZVt0dF0gPSBQcmljZVt0dC0xXSArIG5vaXNlW3R0XQogIH0KfQpmYWN0b3I8LVByaWNlICsgcnVuaWYodCwxLDgpXjIgIyA8PAoKbGlicmFyeShhbmltYXRpb24pCmxpYnJhcnkodGlkeXZlcnNlKQphbmkub3B0aW9ucyhubWF4ID0gMjAsIGludGVydmFsID0gMC4wNSkKdGliYmxlKFByaWNlPVByaWNlLGZhY3Rvcj1mYWN0b3IpIC0+IGRmCiAgYm9vdC5sb3dlc3MoeSA9ZGYkUHJpY2UseD1kZiRmYWN0b3IsIGY9Mi8zLGl0ZXIgPSAxMCxsaW5lLmNvbCA9ICJyZWQiLAogICAgICAgICAgICAgIHhsYWIgPSAiZmFjdG9yIiwgeWxhYiA9ICJQcmljZSIpICU+JQogIHNhdmVHSUYobW92aWUubmFtZSA9ICJsb3dlc3NfYW5pLmdpZiIpCgpgYGAKCiMgSHlwb3RoZXNpcyB0ZXN0aW5nCmBgYHtyfQpsaWJyYXJ5KGF0aSkKIyAgaW5zdGFsbCBhdGkgZnJvbSBteSBnaXRodWIgdXNpbmcKIyByZW1vdmUucGFja2FnZXMoJ2F0aScpCiMgLnJzLnJlc3RhcnRSKCkKIyByZW1vdGVzOjppbnN0YWxsX2dpdGh1YigiYmFycnlxdWlubjEvYXRpIikKYXRpOjpkYWlseV9mYWN0b3JzLT51a19mYWN0b3JzCnVrX2ZhY3RvcnMgJT4lIG11dGF0ZShCdWxsPWlmX2Vsc2Uocm0+MCwiVXAiLCJOb3QgVXAiKSkgLT4gdWtfZmFjdG9ycwptZWFuVVA8LW1lYW4odWtfZmFjdG9ycyRobWxbdWtfZmFjdG9ycyRCdWxsPT0nVXAnXSkKbWVhbl9ub3RVUDwtbWVhbih1a19mYWN0b3JzJGhtbFt1a19mYWN0b3JzJEJ1bGw9PSdOb3QgVXAnXSkKYGBgCgotIFRoZXJlIGhhcyBhbHNvIGJlZW4gYSBtYXJjaCBvZiBtZXRob2RvbG9neSBhbmQgaW5mZXJlbmNlIGZvciBoeXBvdGhlc2lzIHRlc3RpbmcgcmF0aGVyIHRoYW4gZXN0aW1hdGlvbgotIENvbnNpZGVyIHF1ZXN0aW9ucyBhYm91dCB3aGVyZSB2YWx1ZSBpbnZlc3RpbmcgaXMgYmV0dGVyIGluIGEgYnVsbCBvciBhIGJlYXIgbWFya2V0PwotIFRoZSBwbG90cyBzaG93cyB0aGUgaGlzdG9ncmFtIG9mIHRoZXNlIHR3byBncm91cHMgaW4gdGhlIHVrIGRhdGEuVXAgbWFya2V0cyB0byBzZWVtIHRvIGhhdmUgYSBtb3JlIHBvc2l0aXZlIHJldHVybiB0aGVuIGRvd24gbWFya2V0cy4KLSBUaGUgbWVhbiB2YWx1ZXMgZm9yIHVwIGFuZCBub3QgdXAgYXJlIGluIGZhY3QgYHIgc3ByaW50ZigiJS41ZiIsbWVhblVQKWAgYW5kIGByIHNwcmludGYoIiUuNWYiLG1lYW5fbm90VVApYCByZXNwZWN0aXZlbHkgCi0gSXMgdGhlIHBlcmNlaXZlZCBkaWZmZXJlbmNlIGdlbnVpbmUgb3IgYXMgc29tZSBwZW9wbGUgd291bGQgc2F5ICoqYSBzdGF0aXN0aWNhbCBmbHVrZSoqCgpgYGB7ciwgZmlnLndpZHRoPTh9CnVrX2ZhY3RvcnMgJT4lIGdncGxvdChhZXMoeD1obWwsIGZpbGw9QnVsbCkpICsKICBnZW9tX2hpc3RvZ3JhbShiaW5zPTE1KSArCiAgZmFjZXRfd3JhcCh+QnVsbCkKYGBgCgotIFRoZSBjbGFzc2ljIGFuc3dlciB0byB0aGlzIHF1ZXN0aW9uIGlzIHZpYSBhIHR3by1zYW1wbGUgdC10ZXN0CiQkXGZyYWN7XGJhcntWYWx1ZV97VVB9fS1cYmFye1ZhbHVlX3tOb3QgVXB9fX17XGhhdHtzZH19JCQKLSB3aGVyZSAkXGhhdHtzZH0kIGlzIGVzdGltYXRlIG9mIHRoZSBudW1lcmF0b3JzIHN0YW5kYXJkIGRldmlhdGlvbgoKLSAuY29udGVudC1ib3gtYmx1ZVtEaXZpZGluZyBieSBzZCBhbGxvd3MgdXMgKHVuZGVyIEdhdXNzaWFuIGFzc3VtcHRpb25zKSB0byBjb21wYXJlIHRoZSBvYnNlcnZlZCB2YWx1ZSBvZiB0IHdpdGggYSBzdGFuZGFyZCAqKm51bGwqKiBkaXN0cmlidXRpb24sIGluIHRoaXMgY2FzZSBhIFN0dWRlbnTigJlzIHQgZGlzdHJpYnV0aW9uIHdpdGggYHIgbnJvdyh1a19mYWN0b3JzKS0xYCBkZWdyZWVzIG9mIGZyZWVkb20uXQoKCiMjIFR3byBzYW1wbGUgdC10ZXN0IGluZmVyZW5jZQoKYGBge3J9CnQudGVzdCh1a19mYWN0b3JzJGhtbFt1a19mYWN0b3JzJEJ1bGw9PSdVcCddLHVrX2ZhY3RvcnMkaG1sW3VrX2ZhY3RvcnMkQnVsbCE9J1VwJ10pCmBgYAoKLSBXZSBvYnRhaW4gdD0gNS4wNyB3aGljaCB3b3VsZCBjbGFzc2ljYWxseSBiZSBjb25zaWRlcmVkIHZlcnkgc3Ryb25nIGV2aWRlbmNlIHRoYXQgdGhlIGFwcGFyZW50IGRpZmZlcmVuY2UgaXMgZ2VudWluZTsgaW4gc3RhbmRhcmQgdGVybWlub2xvZ3ksIOKAnHdpdGggdHdvLXNpZGVkIHNpZ25pZmljYW5jZSBsZXZlbCAwLjAwMDAwMDMu4oCdCi0gQSBzbWFsbCBzaWduaWZpY2FuY2UgbGV2ZWwgKG9yIOKAnHAtdmFsdWXigJ0pIGlzIGEgc3RhdGVtZW50IG9mIHN0YXRpc3RpY2FsIHN1cnByaXNlOiBzb21ldGhpbmcgdmVyeSB1bnVzdWFsIGhhcyBoYXBwZW5lZCBpZiBpbiBmYWN0IHRoZXJlIGlzIG5vIGRpZmZlcmVuY2UgaW4gcmV0dXJucyBvZiB0aGUgdmFsdWUgZmFjdG9yIGluIHVwIGFuZCBkb3duIG1hcmtldHMKLSBXZSBhcmUgbGVzcyBzdXJwcmlzZWQgYnkgdD01LjA3IGlmIHRoaXMgY29tcGFyaXNvbiBvZiBtZWFuIHJldHVybnMgaXMganVzdCBvbmUgY2FuZGlkYXRlIG91dCBvZiB0aG91c2FuZHMgdGhhdCBtaWdodCBoYXZlIHByb2R1Y2VkIOKAnGludGVyZXN0aW5n4oCdIHJlc3VsdHMuCi0gRm9yIGV4YW1wbGUgaWYgSSB0YWtlIGEgZGlmZmVyZW50IHNldCBzYW1wbGUgcGVyaW9kIGFuZCBydW4gdGhlIHRlc3QgYWdhaW4sIG9yIEkgc3BsaXQgdGhlIHNhbXBsZSBpbnRvIDUgc3ViLXNhbXBsZXMgYW5kIHJ1biB0aGUgdGVzdCBhZ2Fpbi4KCiMjIFRyYWRpdGlvbmFsIGh5cG90aGVzaXMgdGVzdGluZyBhbmQgZmFsc2UgZGlzY292ZXJ5IHRoZW9yeQoKLSBBIHByaW1hcnkgZ29hbCBvZiBlbXBpcmljYWwgcmVzZWFyY2ggaXMgdG8gaWRlbnRpZnkgdGhlIHZhcmlhYmxlcyBpbnZvbHZlZCBpbiBhIHBoZW5vbWVub24KLSBUaGUgaWRlbnRpZmljYXRpb24gb2YgdGhlc2UgdmFyaWFibGVzIGlzIGEgcHJlcmVxdWlzaXRlIHRvIHRoZSBmb3JtdWxhdGlvbiBvZiBhIHRoZW9yeQotIEluIGNsYXNzaWNhbCBzdGF0aXN0aWNzIChlLmcuLCBFY29ub21ldHJpYyksIHRoZSBzaWduaWZpY2FuY2Ugb2YgdmFyaWFibGVzIGlzIGVzdGFibGlzaGVkIHRocm91Z2ggcCB2YWx1ZXMKLSAgcCB2YWx1ZXMgc3VmZmVyIGZyb20gbXVsdGlwbGUgZmxhd3MsIHdoaWNoIGhhdmUgbGVkIHRvIHRoZSBhY2tub3dsZWRnbWVudCB0aGF0IFttb3N0IGRpc2NvdmVyaWVzIGluIGZpbmFuY2UgYXJlIGZhbHNlXShodHRwOi8vZHguZG9pLm9yZy8xMC4xMTExL2pvZmkuMTI1MzApCi0gRmFsc2UgZGlzY292ZXJ5IHJhdGUgdGhlb3J5IGlzIGFuIGltcHJlc3NpdmUgYWR2YW5jZSBpbiBzdGF0aXN0aWNhbCBpbmZlcmVuY2UsIGluY29ycG9yYXRpbmcgQmF5ZXNpYW4gLGZyZXF1ZW50aXN0LCBhbmQgZW1waXJpY2FsIEJheWVzaWFuIGVsZW1lbnRzLgotIEl0IHdhcyBhICpuZWNlc3NhcnkqIGFkdmFuY2UgaW4gYSBzY2llbnRpZmljIHdvcmxkIHdoZXJlIGNvbXB1dGVyLWJhc2VkIHRlY2hub2xvZ3kgcm91dGluZWx5IHByZXNlbnRzIHRob3VzYW5kcyBvZiBjb21wYXJpc29ucyB0byBiZSBldmFsdWF0ZWQgYXQgb25jZS4KCgojIyBTaW11bGF0ZWQgZXhhbXBsZSBvZiBmYWxzZSBkaXNjb3ZlcnkgcHJvYmxlbQoKYGBge3IsZWNobz1UUlVFfQpzZXQuc2VlZCgxMjM1KQpyZXM8LXRpYmJsZSh0ZXN0bm89MToyMCwgcHZhbHVlPTEpCmZvciAoaSBpbiAxOjIwKSB7CiAgVXA9cm5vcm0oMTAwMCkKICBOb1VwPXJub3JtKDEwMDApCiAgcmVzW2ksMl08LXQudGVzdChVcCxOb1VwKVsncC52YWx1ZSddCn0KYGBgCi0gVGhpcyBjb2RlIGZha2VzIHRoZSByZXR1cm4gdXAgdmVyc3VzIGRvd24gcXVlc3Rpb24gZWFybGllciwgd2hlcmUgdGhlICpncm91bmQgdHJ1dGgqIGlzIHRoYXQgdGhlcmUgaXMgbm8gcmVsYXRpb25zaGlwLgotIFRoaXMgaW1wbGllcyB0aGF0IG91ciBzaWduaWZpY2FudCByZXN1bHRzLCB3aGVuIHVzaW5nIGEgdC10ZXN0LCBtdXN0IGJlIGZha2UKLSBXZSByZXBlYXQgdGhlIHRlc3QgMjAgdGltZXMgdXNpbmcgYSBzaW1wbGUgZm9yIGxvb3AuCmBgYHtyIGV2YWw9cmVxdWlyZSgnRFQnKX0KRFQ6OmRhdGF0YWJsZSgKICByZXMsCiAgZmlsbENvbnRhaW5lciA9IEZBTFNFLCBvcHRpb25zID0gbGlzdChwYWdlTGVuZ3RoID0gNSxvcmRlciA9IGxpc3QoMiwgJ2FzYycpKQopCmBgYAotIEFycmFuZ2luZyB0aGUgcmVzdWx0cyBpbiBhc2NlbmRpbmcgb3JkZXIgb2YgcC12YWx1ZXMuCi0gV2l0aCBlbm91Z2ggaHlwb3RoZXNpcyB0ZXN0IHdlIHdpbGwgYWx3YXlzIGRpc2NvdmVyeSBhIGZha2UgcmVzdWx0cy4KLSBJbiBmaW5hbmNpYWwgbWFjaGluZSBsZWFybmluZywgd2hlcmUgd2UgaGF2ZSB0byB3b3JrIHdpdGggaGlnaC1kaW1lbnNpb25hbCBkYXRhIHdpdGggbWFueSBmZWF0dXJlcywgMTAwMCdzIG9mIGNvbXBhcmlzb25zIGFyZSBjb21tb24uIAo=